CPM: A large-scale generative Chinese Pre-trained language model

Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570 GB training data, drew a lot of attention due the capacity few-shot (even zero-shot) learning. However, applying GPT-3 address Chinese tasks is still challenging, as corpus primarily English, are not publicly available. In this technical report, we release Model (CPM) generative pre-training on large-scale data. To best our knowledge, CPM, 2.6 100 largest pre-trained language model, which could facilitate several tasks, such conversation, essay generation, cloze test, understanding. Extensive experiments demonstrate that CPM achieves strong performance many in settings The code available at https://github.com/TsinghuaAI/CPM.